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ABSTRACT 



This paper examines the role played by an office of 
institutional research in developing a new student evaluation of teaching 
protocol. At Southeastern Louisiana University, a comprehensive 4 -year public 
institution, the administration appointed a campus -wide committee to study 
the student evaluations of teaching and to make recommendations for 
improvement. Nine aspects of the process were examined, including the 
philosophy behind the evaluations; prior years’ evaluations; computerized 
data reporting; confidentiality issues; administrative and personal use of 
results; whether or not evaluations were mandatory; and evaluation of 
nontraditional classes. In fall 1995 a new evaluation instrument was 
pilot- tested, refined, and administered to 43 class sections (n=l,100), 
following which interviews were conducted with five classes. Following 
distribution of pilot data, six faculty members were interviewed in depth. 

The final report was presented to the faculty committee charged with 
developing the final instrument, which was developed with the help of 
analyses provided by the institutional research department. In fall 1997, 
implementation of the student evaluations of teaching program was transferred 
to the institutional research office, thus ensuring continuing quality of the 
instrument. Appended are the pilot-test questionnaire, a response form, 
interview protocols, suggestions for evaluating pilot data, and copies of the 
current instrument. (CH) 



******************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 



00 

00 

<N 

<N 

s 



The Role of Institutional Research 

in 

Student Evaluations of Teaching 



Dr. Michelle Hall 
Research Associate 

Office of Institutional Research and Assessment 
Southeastern Louisiana University 
mhall@selu.edu 



Dr. Beatrice Baldwin 

Assistant Vice President for Institutional Planning, Research and Policy Analysis 
Office of Institutional Research and Assessment 
Southeastern Louisiana University 
bbaldwin@selu.edu 



www.selu.edu/Administration/Inst-Research 




Paper presented at the 
1998 Association for Institutional Research 



Minneapolis, MN 
May 1998 

PERMISSION TO REPRODUCE AND j 

DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

AIR 1 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



nFPARTMENT OF EDUCATION 
Sca^nal Research and Implement 
piONAL RESOURCES INFORMATION 
CENTER (ERIC) 

document has been reproduced as 
ved from the person or organization 
nating it. 

>r changes have been made to 
ove reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. __ y 



A*=! 

for Management Research, Policy Analysis, andPlanning 



This paper was presented at the Thirty-Eighth Annual Forum 
of the Association for Institutional Research held in 
Minneapolis, Minnesota, May 17-20, 1998. 

This paper was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality and of 
interest to others concerned with the research of higher 
education. It has therefore been selected to be included 
in the ERIC Collection of AIR Forum Papers. 



Dolores Vura 
Editor 

AIR Forum Publications 




3 



The Role of Institutional Research in Student Evaluations of Teaching 

Abstract: 

One of the most critical issues for faculty at any university is student evaluation of 
teaching. Pragmatically, student evaluations can impact faculty tenure, promotion, salary, 
merit pay, teaching assignments, and collegial professional regard. This paper will 
explore the roles that an office of institutional research can play in developing a new 
student evaluation of teaching system. The paper will recount the experiences of an office 
in serving as technical consultants for the design and gathering of pilot test data; in 
serving as analysts for instrument reliability and validation; and in serving as resources 



for policy discussion. 



History of Revisions to Student Evaluations of Teaching at SLU 



Southeastern Louisiana University is a comprehensive four-year public university which 
has seen a dramatic increase in enrollment over the past decade. The university’s primary focus 
is teaching. Thus, while research productivity is considered, teaching is the major basis for 
hiring and promotion of faculty. Since the late 1980's, SLU has used a Likert scale in student 
evaluations of teaching. The Student Opinion of Teaching (SOT) program has two primary 
objectives. The first is as a mechanism to provide feedback to faculty members in their efforts to 
improve classroom teaching. The second objective is as a tool to be used in 
merit/tenure/promotion decisions. Due to the nature of the institution, the SOT program plays a 
prominent role in assessing teaching effectiveness. 

For a number of years, the University had administered the SOT program utilizing a ten- 
item scale. In the early 1990's, the Faculty Senate raised questions about the usefulness of the 
information provided from the evaluations. At that time only data on the item means was 
provided, and faculty felt that other information would also be useful. Further investigation by 
the Faculty Senate found that there was no written comprehensive policy concerning the 
origination or use of student evaluations of teaching. Thus, the Faculty Senate passed a 
resolution asking the administration to appoint a campus-wide committee to study student 
evaluations of teaching and make recommendations on how to improve them. 

This resolution suggested that nine aspects of the student evaluation process in particular 
be examined. These aspects were: 

• The philosophy or purposes for the existence of the evaluations and the reasons 



for gathering this data; 



• Computer analysis and storage of evaluations from previous years; 

• Evaluations of classes held during the summer session; 

• Computerized reporting of data which would include measures of central 
tendency and variability; 

• Issues of confidentiality, especially for off-campus courses; 

• Administrative and personal uses of the results; 

• Whether or not the evaluations were mandatory or optional, and to which classes 
this applied; 

• Evaluation of non-traditional classes (e.g. team-taught courses, telecourses); and 

• The use of the evaluations as only one of several evaluative measures. 

In Fall 1993, the provost appointed a committee to examine these issues and provide 
recommendations. This committee was composed of members representing the five colleges 
within the University, the Faculty Senate, the SGA, and the Office of Testing. The committee 
was chaired by the Director of Outcomes Assessment. The Provost gave the committee the 
charge of reviewing and studying all aspects of the student evaluation of teaching program and 
making recommendations for revisions and refinements. 

This committee started by reviewing the literature, looking at models used by other 
universities, and collecting input from a wide range of faculty. In the first year of existence, the 
committee developed a working definition of teaching effectiveness, developed a preliminary list 
of behaviors which exemplified the definition of teaching effectiveness, collected input from 
faculty about their concerns with student evaluations of teaching, and recommended a more 
detailed reporting of results. The new components in the reporting of results included item 




2 



V 



frequencies, standard deviations, and the number of students enrolled vs the number of 
completed instruments. 

The committee then divided into two sub-groups. One sub-group worked on developing 
detailed policies and procedures for the administration of student evaluations of teaching, which 
included policies regarding the storage and destruction of data collected. The second 
subcommittee worked on instrumentation. They developed a 63-item instrument to be pilot 
tested. 

Pilot Testing 

In the Fall of 1995, the Office of Institutional Research and Assessment was contacted to 
provide the technical expertise to pilot test two instruments, a quantitative, scaled instrument and 
a qualitative instrument. The faculty committee had developed a scale which identified five 
different behavioral domains within the evaluation of teaching. This domains were Planning and 
Management, Classroom Environment (Acceptance and Openness), Enhancement of Learning, 
Evaluation of Student Progress, and Types of Learning. The pilot instrument consisted of eleven 
items in the area of Planning and Management, eight items in the area of Classroom 
Environment, eighteen items in the area of Enhancement of Learning, ten items in the area of 
Evaluation of Student Progress, ten items in the area of Types of Learning, six general items, and 
four demographic items. Also included in the pilot test was a narrative response form which 
asked students to comment on the strengths of the course, suggestions for the course , and 
suggestions for the instructor . A complete copy of the pilot instruments can be found in 
Appendices A and B. 
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In working with the committee, Institutional Research decided that a two-pronged 
approach to the pilot testing would provide the most valuable data. A traditional pilot test was 
conducted by giving the large set of items to a sample of classes and analyzing the results 
according to traditional psychometric practice. Furthermore, interviews with students, faculty 
and department heads were conducted to obtain further information on the validity of the 
instrument, the clarity of directions, policy issues, and the ease of administration. Table 1 
outlines the timeline followed for the pilot test. 



Table 1 

Timeline for SOT Pilot Test 



Dates 


Activity 


November 13 th - November 17 th , 1995 


Collected pilot data in selected sample of classes 


November 20 th - November 24 th , 1995 


Conducted group interviews with students 


November - End of Semester, 1995 


Analysis of quantitative data 


Spring Semester Between l sl and 3 rd Week, 1996 


Pilot SOT Reports and narratives sent to faculty 
and Department Heads who participated 


Spring Semester Between 4 th and 5 th Weeks, 1996 


Conducted individual interviews with selected 
faculty and Department Heads 


March 1996 


Presented report to faculty committee 


March - End of Spring Semester, 1996 


Did further analyses. Committee reviewed and 
discussed report in order to make recommendations 


Summer 1996 


University Academic Affairs Council reviewed and 
approved changes in forms and procedures for use 
in Fall 1996 


Fall 1996 


New instrument and procedures implemented for 
the first time 



Throughout the entire pilot test, the Office of Institutional Research and Assessment 
considered the Student Opinion of Teaching instrument and the instructional evaluation process 
as vitally important to SLU students and faculty, not only in terms of improving teaching and 
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learning but also in terms of making important decisions about tenure, promotion, and merit. 
Thus the office attempted to make the pilot testing process as technically sound as possible, 
while allowing for instructors to volunteer for participation and controlling for the confidentiality 
of results. The use of volunteer instructors was necessary to the process, as participating in the 
pilot involved giving up class time at the end of the semester. For those classes which 
participated, the SOTs were administered twice. In addition, some classes had to give up class 
time so that an Institutional Research staff member could come in and interview the class. It 
should be noted that the Institutional Research office, being external to the SOT committee, 
provided an increased measure of process validity in that the office did not have input into the 
instruments nor did the office make substantive recommendations for change after the pilot. 
Participants 

A stratified sampling procedure identified a target population of large lecture classes, 
small lecture classes, and lab/activity classes, further stratified by level (undergraduate and 
graduate) and the tenure status of the instructor for a total of 12 combinations of class type, level 
and instructor type. In the fall semester, department heads were contacted to suggest target 
sample classes in the 12 categories. Department Heads were informed up front that participation 
in the pilot would probably consist of: 

• administering the new instrument (Likert-scaled items and open-ended narrative 
form questions) as designed by the SOT Committee to students in selected 
classes; 
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• allowing an Institutional Research and Assessment staff member to do a short 
group interview with selected classes to get feedback on administration and 
readability; 

• allowing an Institutional Research and Assessment staff member to do a 
confidential interview with the instructor (small purposive sample) after the SOT 
results had been distributed in January; and 

• allowing an Institutional Research and Assessment staff member to do a 
confidential interview with the department head (small purposive sample) on the 
utility of the new instrument. 

After suggesting classes for participation, department heads were asked to verify with those 
faculty members affected that the pilot testing would not pose a problem with the class schedule. 
The selected sample consisted of 43 sections with a projected enrollment of over 1,100 students. 
Table 2 shows how many sections were in each of the stratification levels. 

Table 2 



Stratification of Target Sample 





Large lecture 


Small lecture 


Activity /lab 


Faculty Status 


UG 


G 


UG 


G 


UG 


G 


Tenured 


4 


3 


5 


3 


2 


0 


Tenure-Track 


4 


1 


4 


3 


2 


1 


Other 


5 


0 


2 


0 


3 


0 
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Method 



Following the distribution of the then current instrument, 43 class sections (n= 1 100) were 
administered the pilot instrument. Following the administration of the pilot, a purposive sample 
of six classes of students were selected to be interviewed regarding the pilot. Due to 
circumstances beyond the researcher’s control (the instructor became ill and canceled class), one 
of the interviews did not take place. Two of the five classes interviewed were graduate classes, 
while three were undergraduate classes. Three of the instructors were non-tenured, tenure track, 
and two were tenured. Three of the classes were small lecture classes, and two were activity/lab 
classes. Large lecture classes were purposively left out of the interviews due to the problems 
inherent in interviewing a large number of students at once. These interviews covered students’ 
opinions regarding the relevance, importance and understanding of the items (for both the Likert- 
type items and the open-ended items), opinion of the procedure for administering the instrument, 
and comparing the pilot version to the current version. A copy of the protocol followed for 
interviewing students (and the protocols for faculty and department heads) can be found in 
Appendix C. 

Following the distribution of preliminary reports based on the pilot test data, six faculty 
members were interviewed in-depth. Two of the faculty were tenured, two were non-tenured, 
tenure track, and two were non-tenure track. The faculty members’ classes which participated in 
the pilot included two graduate classes and four undergraduate classes. Furthermore, three of the 
classes were small lecture, two were large lecture, and one was an activity/lab. These interviews 
covered their reactions to the pilot items (both Likert-type and open-ended items), the reporting 
of the data, comparisons with the current instrument, their plans for using the instrument in 
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course improvement, and reactions they observed from students. A copy of the protocol 



followed for interviewing faculty can be found in Appendix C. 

Also, after the preliminary reports had been distributed, six department heads, 
representing the four Colleges of the University were interviewed in-depth. These interviews 
covered department heads reactions to the pilot items (both Likert-type and open-ended items), 
the reporting of the data, comparisons with the current instrument, their thoughts on being 
provided with a copy of the written comments, concerns regarding the instruments impact on 
tenure and/or promotion, and reactions they observed from students and faculty. A copy of the 
protocol followed for interviewing faculty can be found in Appendix B. 

Summary - Results of the Pilot Test 

Overall, there was a total of 903 participants. Of the total, 22% (n=196) were Freshmen, 
17% (n=151) were Sophomores, 17% (n=153) were Juniors, 21% (n=190) were Seniors, 16% 
(n=143) were Graduate students, 1% (n=l 1) indicated Other, and 7% (n=59) did not indicate 
their class. The vast majority (75%, n=675) indicated that the course was required in their major, 
18% (n=158) were taking the class as an elective, and 7% (n=70) did not give an indication of 
whether or not the class was required. 

Twenty-three percent (23%, n=2 11) indicated they had a cumulative GPA in the range of 
3.50-4.00, 24% (n=2 1 5) in the range of 3.00-3.49, 31% (n=277) in the range of 2.50-2.99, 12% 
(n=l 12) in the range of 2.00-2.49, 2% (n=19) less than 2.00, and 8% (n=69) did not indicate their 
GPA range. Finally, the majority of students expected to get an A in the course (41%, n=373), 
32% (n=292) expected a B, 15% (n=137) expected a C, 3% (n=26) expected a D, and only 1 
student (.1%) expected to fail. The remaining 8% (n=74) did not indicate an expected grade. 
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Table 3 presents a summary table of participants by section. The first column (Sample 
Code) refers to the sampling scheme group. The first number in the code refers to whether the 
class was an undergraduate or graduate level class (1 = undergraduate, 2 = graduate). The second 
number refers to the status of the instructor (1 = tenured, 2 = tenure-track, non-tenured, and 3 = 
non-tenure track). The last number refers to the type of class (1 = a large lecture class [undergrad 
greater than 40 and graduate greater than 20], 2 = small lecture class, and 3 = activity or lab 
class). Thus, for example, 123 refers to an undergraduate laboratory class taught by a tenure 
track, non-tenured faculty member. The second column (Number Enrolled) is the number of 
students enrolled in the class according to official class rosters. The third column (Number of 
Responses) is the number of students who actually completed the quantitative instrument. The 
fourth column (% Responded) is the percent of students officially enrolled who completed the 
quantitative portion. The fifth column (Number of Comments) is the number of narrative forms 
which actually contained written comments. The last column (% of Comments) is the percent of 
students who wrote open-ended comments, based on the number of students who responded to 
the numerical items in each section. 
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Table 3 



Summary of Participants by Section 



Sample Code 


Number 

Enrolled 


Number of 
Responses 


% Responded 


Number of 
Comments 


% Comments 


111 


55 


42 


76.36% 


36 


85.71% 


111 


33 


29 


87.88% 


28 


96.55% 


111 


40 


33 


82.50% 


26 


78.79% 


111 


93 


65 


69.89% 


0 


0.00% 


112 


17 


15 


88.24% 


14 


93.33% 


112 


12 


12 


100.00% 


10 


83.33% 


112 


21 


16 


76.19% 


16 


100.00% 


112 


16 


11 


68.75% 


5 


45.45% 


113 


20 


17 


85.00% 


16 

— V 


74.12% 


113 


20 


13 


65.00% 


11 


84.62% 


121 


39 


31 


79.49% 


28 


90.32% 


121 


32 


27 


84.38% 


19 


70.37% 


121 


32 


23 


71.88% 


17 


73.91% 


121 


35 


33 


94.29% 


18 


54.55% 


122 


29 


23 


79.31% 


19 


82.64% 


122 


20 


16 


80.00% 


14 


87.50% 


122 


20 


12 


60.00% 


12 


100.00% 


123 


16 


15 


93.75% 


11 


73.33% 


123 


15 


14 


93.33% 


11 


78.57% 


123 


33 


29 


87.88% 


21 


72.41% 


131 


32 


29 


90.63% 


24 


82.76% 


131 


32 


18 


56.25% 


16 


88.89% 


131 


47 


38 


80.85% 


29 


76.32% 


131 


38 


23 


60.53% 


17 


73.91% 


131 


63 


42 


66.67% 


37 


88.10% 


132 


24 


20 


83.33% 


20 


100.00% 
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Table 3 cont. 

Summary of Participants by Section 



Sample Code 


Number 

Enrolled 


Number of 
Responses 


% Responded 


Number of 
Comments 


% Comments 


132 


21 


15 


71.43% 


15 


100.00% 


132 


27 


21 


77.78% 


19 


90.48% 


133 


27 


23 


85.19% 


23 


100.00% 


133 


36 


30 


833.33% 


25 


83.33% 


133 


16 


14 


87.50% 


14 


100.00% 


211 


21 


18 


85.71% 


18 


100.00% 


211 


20 


18 


90.00% 


18 


100.00% 


211 


35 


31 


88.57% 


27 


87.10% 


212 


22 


20 


90.91% 


10 


50.00% 


212 


6 


6 


100.00% 


5 


83.33% 


212 


8 


8 


100.00% 


8 


100.00% 


221 


28 


22 


78.57% 


20 


90.91% 


222 


10 


9 


90.00% 


7 


77.78% 


222 


9 


9 


100.00% 


8 


88.89% 


222 


10 


9 


90.00% 


8 


88.89% 


223 


4 


4 


100.00% 


4 


100.00% 


Total: 42 
Sections 


1134 


903 


79.63% 


704 


77.96% 



Report to the Committee 

Institutional Research prepared a report which included a frequency distribution, mean, 
median and standard deviation for each scaled item. Also included in the report were inter-item 
correlations for each of the previously defined behavioral domains as well as the correlations 
between the domains. Also included were Cronbach Alpha Reliability estimates for each domain 
and an exploratory factor analysis. Finally, the report included summaries of all the interviews 
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that were conducted, and suggestions and caveats for interpreting the data and evaluating 
administration and other procedural issues. The concerns and suggestions for interpretation and 
evaluation can be found in Appendix D. 

Institutional Research staff presented this report to the faculty committee charged with 
developing the new instrument and to the Deans’ Council. Included in the presentations were 
suggestions and comments based on the data in regards to policy implications. 

Due to the sensitive nature of the topic, its impact on faculty and academic policy, and its 
role in the process, Institutional Research did not make rigid recommendations regarding which 
items should be included in the new instrument or the policy changes that needed to be made. 
Theses decisions needed to be reached by consensus of the faculty committee. However neither 
did Institutional Research simply present the report and disappear. Institutional Research staff 
continued to work with the committee, providing further analyses, reiterating the limitations of 
the data, and steering the committee away from non-data driven decisions. 

Based on our analyses, the committee developed a quantitative instrument composed of 
twenty-four items and three demographic items. There are five items in the Planning and 
Management domain, five items in the Classroom Environment (Acceptance and Openness) 
domain, six items in the Enhancement of Learning domain, five items in the Evaluation of 
Student Progress domain, and three general items. The committee also recommended that a copy 
of all narrative forms be distributed to the faculty members, their respective department heads 
and deans. Furthermore, the committee recommended that the SOT be given at the beginning of 
each class period rather than the end; that for courses such as practica, internships and 
independent studies, departments develop their own methods of obtaining feedback; that SOTs 
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be administered during the summer semester. In summer 1996, the new Student Opinion of 
Teaching form was approved by the Academic Affairs Council. The narrative form permits 
students to comment on strengths suggestions for improvement for the instructor, the activities 
and tests, the books and materials, and the classroom environment. A copy of the current SOT 
instruments can be found in Appendix E. 

Conclusions 

The new student evaluation of teaching system has been in place for two years. The 
scope and role of Institutional Research in regards to student evaluation of teaching has increased 
over the past two years. When the new instrument was first used, the Office of Testing oversaw 
the administration, storing, and reporting process. In June 1997, the implementation of the 
student evaluations of teaching program was transferred to Institutional Research. The office 
added a new staff person whose main responsibility is overseeing the SOT process. 

There are several benefits to housing the process within the Institutional Research Office. 
One of the benefits is that Institutional Research now has access to the data, and thus can fulfill 
requests from campus constituents such as Deans and the Provost who ask for special analyses of 
aggregate data. For example, the office has done a study for one of the Colleges which examined 
the relationship between student evaluations and grade inflation. The office has done a study for 
another College which looked at whether there was a difference in SOT scores between 
undergraduate classes and graduate courses. 

Housing the process within the Institutional Research Office also ensures that the office 
will have continuing technical input as modifications and changes to the instrument and process 
occur. For any university this input from Institutional Research may be helpful since the 
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psychometric and statistical expertise needed can often be found within the IR office. Another 
benefit may be that the IR office has no vested interest in the processes or outcomes, so the office 
can be viewed as an impartial third party. A final benefit is that as states move further towards 
issues of accountability, the IR office plays a major role, and student evaluations of teaching can 
be an important part of performance indicator systems. 

In the Future 

More instrument research will be necessary to insure the continuing quality of the 
instrument. Institutional Research is not taking sole responsibility for the direction of this 
research, but is working with the faculty committee to help guide the research. In the future the 
office will collect information on faculty perceptions of the instrument as course evaluation tools. 
In addition, surveys of deans and department heads will center on the utility of current reporting 
formats and the policy manual. Furthermore, focus group sessions with selected graduate and 
undergraduate student groups will target perceptions regarding ease of completion and clarity. 

The Institutional Research office is also working with departments offering classes for 
which the current instrument is not appropriate (e.g. independent study, pratica, etc.) by helping 
them develop more appropriate instruments. For example, the office is currently working with 
the School of Nursing to pilot test an instrument for evaluating its clinical labs. The nature of 
these labs is such that the current instrument is clearly not appropriate. Faculty in the School of 
Nursing developed items which they felt were appropriate based on the literature and their own 
experiences. These questions were then brought to the Institutional Research office which then 
advised them on technical aspects of item writing, developed an instrument for them and is 
currently collecting data which will be used to help finalize an instrument to be implemented 
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starting in Fall 1998. As another example, the office has also worked with a Distance Education 
Committee to develop an instrument which is more appropriate to courses which are taught at 
least 50% on the Internet. This instrument is being used this semester for the first time. During 
the summer, the office will work with a committee to develop an instrument which is appropriate 
for use with compressed video courses, and the instrument will be used in Fall 1998 for the first 
time. 

Institutional Research has also worked with the faculty committee and University 
Academic Affairs Council to refine the policy manual which outlines the procedures for 
administering the Student Opinion of Teaching program. Due to the political climate on campus, 
the original decision of the SOT committee to share narratives with department heads and deans 
was not implemented. Recently the Academic Affairs Council reevaluated this decision and has 
approved a pilot program which will allow for students’ narrative comments to be shared with 
the faculty member’s department head. Part of the impetus for this pilot program came from data 
Institutional Research collected during the pilot test of the instrument. Also included in the pilot 
are guidelines for the evaluation of narrative comments. Theses guidelines were developed with 
the assistance of Institutional Research staff. This new program will be implemented in Fall 
1998 and will be reviewed a year later. 

The Institutional Research office also has responsibility for maintaining and updating the 
policy manual for the Student Opinion of Teaching program. This manual has been provided to 
all deans and department heads, and the office is also responsible for maintaining a World Wide 
Web version of the manual. The current manual can be found on the web at: 

www.selu.edu/Administration/Institutional-Research/IE/sotpol.htm 
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The manual is currently in the process of being revised, and revisions to the policy should be 
posted to the web by mid-summer. 

Within the next year, Institutional Research will also begin working with a newly created 
faculty development office, the Center for Faculty Excellence. This new office within the 
division of Academic Affairs will provide professional development programs for faculty and 
will concentrate on improving teaching and classroom assessment. Student evaluation of 
teaching results will be aggregated at the college and university level to provide data about the 
teaching performance indicators. Such data can be analyzed by course level, curriculum 
component level (i.e., general education requirement, course in the major, etc.), and faculty rank 
to provide insights about needed professional development programming and objectives. In 
addition, complementary methods of assessing classroom teaching is being discussed on campus, 
and the IR office 

In summary, an office of institutional research should not be reluctant to engage in the 
arena of student teaching evaluations if it can offer qualified technical expertise and sound advice 
on continued research and development of such systems. Indeed, in today’s transformational 
higher education environment, institutional research offices should seek opportunities to provide 
leadership in important areas like the improvement of university teaching. This leadership role 
provides an institutional research office the opportunity to work closely with faculty and campus 
administration, and enter into a dialogue on an issue of importance to the entire campus 
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community. 



Appendix A 



Student Opinion of Teaching Questionnaire 
Pilot Test - Quantitative Form 
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Southeastern Louisiana University 




Student Opinion of Teaching Questionnaire 

(Revised) 

Pilot Test, 

Fall 1995 



To Participating Students: 

This pilot version of the SLU Student Opinion of Teaching questionnaire requires that you use a 
general purpose scanning sheet for your answers. Please be sure to blacken the appropriate 
circles completely. 

On side 1 of the form, fill in the course computer number in the section marked “Identification 
Number.” Also fill in the “Special Codes” section with a number that will be given to you by the 
instructor. You do not need to fill in any other section on the left-hand portion of Side 1 . 

Read each item on the Student Opinion of Teaching Questionnaire carefully and fill in the circles 
under the appropriate columns to indicate your response to each item. Be sure to also complete 
the demographic items at the end of the questionnaire. 

You have also been given a carbonless form for your comments about the course and the 
instructor. Please fill in the instructor’s name and the course computer number at the top of the 
form. Your responses to the questions about the course and the instructor will provide 
information that will be used by the faculty and the department for course and instructional 
improvement. 

You will have approximately 1 5 minutes to complete the forms. When you have finished with 
the forms, please return all materials to the envelope. A student from your class will return the 
materials to the department office. 

Your feedback is important to us! 

Thank you for your participation! 
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Planning and Management 


Never 










Always 


1 . Course learning objectives are clearly stated 


i 


2 


3 


4 


5 


6 


2. My responsibilities and expectations for learning are clearly 
stated. 


i 


2 


3 


4 


5 


6 


3. Course activities are well-organized 


i 


2 


3 


4 


5 


6 


4. Class activities help me achieve the learning objectives of 
the course. 


i 


2 


3 


4 


5 


6 


5. Out-of-class assignments help me achieve course 
objectives. 


i 


2 


3 


4 


5 


6 


6. Out-of-class assignments are integrated with class activities 


i 


2 


3 


4 


5 


6 


7. Learning objectives for each class are clearlv stated 


i 


2 


3 


4 


5 


6 


8. The instructor is well-prepared for class 


i 


2 


3 


4 


5 


6 


9. The instructor begins and ends class on time 


i 


2 


3 


4 


5 


6 


10. The instructor uses time efficiently for teaching and 
learning 


i 


2 


3 


4 


5 


6 


1 1 . The instructor uses learning activities that motivate me to 
learn 


i 


2 


3 


4 


5 


6 


Classroom Environment 


12. The classroom environment contributes to my learning 


i 


2 


3 


4 


5 


6 


13. The instructor maintains an atmosphere of mutual courtesy 


i 


2 


3 


4 


5 


6 


14. The instructor is enthusiastic about teaching this course 


i 


2 


3 


4 


5 


6 


15. The instructor is enthusiastic about mv learning in this 
course 


i 


2 


3 


4 


5 


6 


16. The instructor demonstrates interest in my leaming/progress 
in this course 


i 


2 


3 


4 


5 


6 


17. The instructor is sensitive to my needs and feelings 


i 


2 


3 


4 


5 


6 


18. The instructor is willing to provide help and guidance 
outside of class 


i 


2 


3 


4 


5 


6 


19. I am encouraged to participate in class discussions/activities 


i 


2 


3 


4 


5 


6 


Enhancement of Learning 


20. The instructor’s speech is clear and easy to understand 


i 


2 


3 


4 


5 


6 


21 . The instructor arouses and maintains my interest in the 
subject 


i 


2 


3 


4 


5 


6 


22. Teaching methods stimulate my involvement in learning 


i 


2 


3 


4 


5 


6 


23. Course content is at an appropriate level of difficulty 


i 


2 


3 


4 


5 


6 


24. The instructor makes clear all directions and explanations 
related to course content. 


i 


2 


3 


4 


5 


6 
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Never 










Always 


25. The instructor assists me in organizing information 


1 


2 


3 


4 


5 


6 


26. The instructor clearly explains difficult material 


1 


2 


3 


4 


5 


6 


27. The instructor helps me understand relationships among the 
topics in the course 


1 


2 


3 


4 


5 


6 


28. The instructor summarizes topics before moving to new 
topics 


1 


2 


3 


4 


5 


6 


29. The instructor summarizes important topics or ideas at the 
end of class 


1 


2 


3 


4 


5 


6 


30. The instructor clearly answers students’ questions 


1 


2 


3 


4 


5 


6 


3 1 . Te instructor carries out learning activities at an appropriate 
pace 


1 


2 


3 


4 


5 


6 


32. The instructor asks thought-provoking questions 


1 


2 


3 


4 


5 


6 


33. The instructor asks questions that allow me to compare and 
contrast ideas 


1 


2 


3 


4 


5 


6 


34. The instructor realizes when students are confused and 
clarifies as needed 


1 


2 


3 


4 


5 


6 


35. I receive periodic feedback about my learning progress 


1 


2 


3 


4 


5 


6 


36. Teaching aids (e.g., audio-visual material) are used to 
enhance my learning 


1 


2 


3 


4 


5 


6 


37. I am encourage to apply course content to real life situations 
and/or practice 


1 


2 


3 


4 


5 


6 


Evaluation of Student Progress 


38. The instructor has made clear the basis for grading 


1 


2 


3 


4 


5 


6 


39. The instructor has an appropriate balance among tests, 
assignments, projects, etc. 


1 


2 


3 


4 


5 


6 


40. Tests and other evaluations of my learning reflect the 
course content 


1 


2 


3 


4 


5 


6 


41 . Test items are clearly written 


1 


2 


3 


4 


5 


6 


42. Tests are of reasonable length 


1 


2 


3 


4 


5 


6 


43. Tests/other evaluations provide a fair chance to demonstrate 
achievement 


1 


2 


3 


4 


5 


6 


44. The instructor grades assignments and tests fairly 


1 


2 


3 


4 


5 


6 


45. The instructor returns assignments/tests within a reasonable 
length of time 


1 


2 


3 


4 


5 


6 


46. I receive sufficient feedback on all graded work 


1 


2 


3 


4 


5 


6 


47. I know where I stand academically during the course 


1 


2 


3 


4 


5 


6 
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Tvpes of Learning 




Never 










Always 


48. The instructor emphasizes learning factual information 


1 


2 


3 


4 


5 


6 


49. The instructor encourages students to help develop 
concepts 


1 


2 


3 


4 


5 


6 


50. The instructor actively involves me in 

understanding/applying principles and rules 


1 


2 


3 


4 


5 


6 


51. The instructor actively involves me in understanding and 
applying theories 


1 


2 


3 


4 


5 


6 


52. The instructor actively involves me in using problem- 
solving skills 


1 


2 


3 


4 


5 


6 


53. The instructor encourages me to think creatively 


1 


2 


3 


4 


5 


6 


54. Course activities enhance my knowledge of self and others 


1 


2 


3 


4 


5 


6 


55. Course activities enhance my professional, career, and job- 
related skills 


1 


2 


3 


4 


5 


6 


56. Course activities enhance the development of my writing 
skills 


1 


2 


3 


4 


5 


6 


57. Course activities enhance the development of my speaking 
skills 


1 


2 


3 


4 


5 


6 


In Mv Opinion... 


Poor 










Excellent 


58. The quality of instruction was... 


1 


2 


3 


4 


5 


6 


59. Contribution of the course to my learning was... 


1 


2 


3 


4 


5 


6 


60. Contribution of the course to my professional preparation 
was... 


1 


2 


3 


4 


5 


6 


61. Overall value of the course was... 


1 


2 


3 


4 


5 


6 


62. Chances I would choose to take this instructor again are... 


1 


2 


3 


4 


5 


6 


63. Books and materials were... 


1 


2 


3 


4 


5 


6 



Student Demographics 



64. My student classification is l=Freshmen 

2=Sophomore 

3=Junior 

4=Senior 

5=Graduate 

6=Other 

65. My cumulative GPA is 1=3.50-4.00 

2=3.00-3.49 
3=2.50-2.99 
4=2.00-2.49 
5=Less than 2.0 



66. The course is l=Required in my major 

2=An elective in my major 

67. Expected grade for the course is (bubble in A, 
B, C, D, or F) 
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Appendix B 

Student Opinion of Teaching Instrument 
Pilot Narrative Response Form 
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Southeastern Louisiana University 







Student Opinion of Teaching 
Narrative Response Form 



Instructor’s Name: 

Course Computer Number: 

Please write out your comments to the following: 

The strengths of the course were... 

My suggestions for the course are... 

My suggestions for the instructor are... 



White copy — Instructor Yellow copy — Institutional Research and Assessment 
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Appendix C 



Interview Protocols 




25 



28 



Protocol for Interviewing Students 



I’m Michelle Hall from Institutional Research and Assessment. You recently took a new form of 
the Student Opinion of Teaching. It has been expanded to 65 questions, instead of only 10. This 
was done to allow students to more thoroughly express their opinion about the way professors 
teach. As part of testing the new questionnaire, I would like to ask you a few questions about it. 
Please feel free to say how you honestly feel about it, good or bad. Your name will never be 
associated with you comments. This interview will be tape recorded, but that is only so that 
when I am writing the report, I can go back and make sure I am accurately reflecting what you 
said. I would like for everyone to participate in this discussion, all of your opinions are 
important. 

First, I want to ask you some general questions about the procedures. How was the amount of 
time you were given to complete the instrument? Was it too long, not long ? 

Next, how were the instructions? How clear were the instructions? How could the instructions 
be improved? 

What procedure was followed for giving the instrument? What did the professor (instructor) do 
while you completed the questions? How comfortable are you with the manner in which 
administration and collection were handled? Were you allowed the allotted time to complete it? 
Did the professor read the instructions? When did you take it (at start or end of class)? 

Now I want to ask you some questions regarding the content of the instrument. Here are some 
copies of the questions that you answered. What questions were confusing? What was 
confusing about them? Are there questions that you think are asking the same thing? 

Now, on the other hand, what questions or aspects of the class are not addressed? What would 
you like to have added to the questions? 

As far as the open-ended questions, what did you think of those? Were they too broad, too 
narrow, wrong questions? 

How do you feel about your written comments being shared with the professor’s (instructor’s) 
department head? 

Are there any other comments you have about the instrument? 

If you have comments or concerns that you don’t feel like sharing in front of your peers, or that 
you think of later, please write them down and send them to me at the Office of Institutional 
Research and Assessment. Thank you for your time. 
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Protocol for Interviewing Faculty Members 



First, I want to ask you about the reporting of the data. Were you able to understand the 
information you received? Was the information provided useful? Are there other types of 
information you would like presented? Do you have any suggestions for making the information 
more useful for yourself? 

Make sure have copy of SOT. What about the items on the SOT? Do you think they were 
adequate to provide information to help improve your teaching? Were there other areas that you 
believe should have been addressed? Do you believe the questions were appropriate? Were they 
appropriate for the type of class you taught (i.e. large/small, UG/Grad, Activity /lab/lecture)? 
Looking at the SOT, what specific questions do you think are essential? What questions do you 
think are not necessary? 

Moving on to the open-ended questions, do you read all the comments? Do they provide you 
with useful information? Do you think it is beneficial to have a separate form for the comments? 
Are there other questions you think should be asked in addition to, or instead of? 

How do you feel about written comments being shared with your department head? Do you have 
any concerns about doing so? c 

In a more general sense, how have you, or how will you, use the information (both quantitative 
and qualitative) that has been provided? 

How would you compare this instrument to the one previously used for the SOT? Which do you 
prefer? Why? 

What are your concerns regarding the impact of this instrument on tenure and/or promotion? 

What kind of reactions to the pilot did you observe from your students? What kind of comments 
did they make about it? 

Do you have any other comments regarding the SOT pilot? 
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Protocol for Interviewing Department Heads 



First, I want to ask you about the reporting of the data. Were you able to understand the 
information you received? Was the information provided useful? Are there other types of 
information you would like presented? Do you have any suggestions for making the information 
more useful for yourself? 

Make sure have copy of SOT. What about the items on the SOT? Do you think they were 
adequate to provide information to help evaluate and provide feedback for your faculty? Were 
there other areas that you believe should have been addressed? Do you believe the questions 
were appropriate? Do you believe the questions were appropriate? Looking at the SOT, what 
specific questions do you think are essential? What questions do you think are not necessary? 

Moving on to the open-ended questions, did you read all the comments? Do they provide you 
with useful information? Do you think it is beneficial to have a separate form for the comments? 
Are there other questions you think should be asked in addition to, or instead of? 

How do you feel about being provided with the written comments? How do you plan on using 
the written comments? 

o 

In a more general sense, how have you, or how will you, use the information (both quantitative 
and qualitative) that has been provided? 

How would you compare this instrument to the one previously used for the SOT? Which do you 
prefer? Why? 

What are your concerns regarding the impact of this instrument on tenure and/or promotion? 

What kind of reactions to the pilot did you observe from your faculty? What kind of comments 
did they make about it? 

Do you have any other comments regarding the SOT pilot? 
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Appendix D 

Concerns and Suggestions for Interpreting and Evaluating Pilot Data 
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General Conclusions 



• Committee should not rely on quantitative data to make decisions about needed revisions, 
o Is the SOT norm-referenced or criterion-referenced? 

o Ceiling effect of SOT scores. 

o Quantitative and qualitative data provid a more complete picture. 

• There are some basic issues that need to be addressed, 
o Quantitative instrument 

Which items should be included? 

- Does an item contain “educational” jargon that others will not understand? 

What demographic items need to be included? 

Should a Not Applicable response be available? If not, items should be applicable to 
all types of classes, 
o Qualitative (Narrative) Instrument 

Should a separate form be used, or should it be part of the quantitative form? 

What should be included in the qualitative form? 
o Administrative procedures 

What type of security is needed for the data and the forms? 

When should SOTs be administered? At what point in the semester, when during the 
class? 
o Items 

s' 

There are too many items, some need to be deleted. 

Committee should review items in terms of redundancy and applicability. 

30 
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High reliability may be an artifact of pattern scoring and/or redundancy. 

- All correlations are statistically significant at p=.01 level, an artifact of large sample 
size. 

The factor analysis does not confirm the dimensionality of the SOT rating scale. 

• Policy Issues 

o What type of standards need to be developed for using the SOT results in 
promotion/tenure/merit issues? 

o What type of tiaining, if any, need to be presented to enforce the standards? 

• Interview sessions need to be carefully read as they contain many recommendations for 
administration and other procedural issues. 
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Suggested Action Plan 



1 . Review research on student evaluations of teaching 

2. Determine the optimum number of items 

3. Critically evaluate all interview responses 

4. Interpret the factors - use or discard them 

5. Use numerical data analysis to evaluate redundancy 

6. Select items 

7. Develop policies for administration and protocol (including usage standards and training) 

8. Send all recommendations to the Provost 
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Appendix E 



Current Student Opinion of Teaching Instruments 
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SOUTHEASTERN LOUISIANA UNIVERSITY 
Student Opinion 



Directions : 

o This instrument is used by faculty and administrators to evaluate and improve 
the effectiveness of SLU’s instructors and courses. 

o it is very important to completely and accurately fill in the four-digit computer 
number. If the box is not filled in accurately, your opinion will not be provided to 
your instructor. If your class is taught by more than one instructor, fill in the 
instructor’s number in the team-taught box. If your class is taught by only one 
instructor, please leave the team-taught box blank. 

o Read each item carefully and fill in the circle under the appropriate column to 
indicate your response (using a number 2 pencil). Please note that a “6" on the 
right side indicates you “Strongly Agree” with the statement, and a “ 1” on the 
left side indicates you “Strongly Disagree" with the statement. Complete both 
sides of the instrument. 

o The instructor should read the instructions to the class, assign a student to 
deliver the questionnaires to the designated location, and allow at least 15 
minutes for completion. This evaluation process should be conducted at the 
beginning of the class period and the instructor should leave the room after 
reading the instructions. Please report any deviation from these procedures to 
the Dean of the College offering the course. Contact the Office of Institutional 
Research and Assessment (549-3639) for the name of the appropriate Dean. 



Compute? 

Mamba? 











® © © © 
© 00.0 
©©©© 
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©©©© 
©©©© 
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Team-saught 

Number 



0 

O 

© 

© 

© 

© 

© 

© 

© 

m 



2 


<& 

/ 

/ 






% 


V 


K 






i 


2 


3 


4 


5 


6 


— 


Planning and Management 


1 












— 


1. Course activities are well organized. 


© 


© 


© 


© 


© 


© 


— 


2. The instructor is well prepared for class. 


© 


© 


© 


© 


© 


© 


— 


3. The instructor begins and ends class on time. 


© 


© 


© 


© 


© 


© 


— 


4. The instructor avoids class cancellations. 


© 


© 


© 


© 


© 


© 




5. The instructor uses time efficiently for teaching and learning. 


© 


© 


© 


© 


© 


© 


I 


Classroom Environment (Acceptance and Openness) 
















6. The instructor maintains an atmosphere of courtesy and respect in class. 


© 


© 


© 


© 


© 


© 




7. The instructor is enthusiastic about teaching this course. 


© 


© 


© 


© 


© 


© 


a 


8. The instructor’s speech is clear and understandable. 


© 


© 


© 


© 


© 


© 




9. The instructor is willing to provide help and guidance outside of class. 


© 


© 


© 


© 


© 


© 




10. The instructor is open to student questions. 


r © 


© 


© 


© 


© 


© 




Enhancement of Learning 














— 


1 1. Course learning objectives are clear. 


© 


© 


© 


© 


© 


© 


■i 


12. The instructor attempts to interest me in the subject. 


© 


© 


© 


© 


© 


© 


— 


13. The instructor helps me understand relationships among topics in the course. 


© 


© 


© 


© 


© 


© 




14. The instructor summarizes important topics or ideas. 


© 


© 


© 


© 


© 


© 




15. The instructor encourages me to relate course content to real life situations 


© 


© 


© 


© 


© 


© 




and/or practice. 
















16. The instructor encourages critical thinking. 


© 


©. 


© 


© 


© 


© 
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Evaluation of Student Progress 


i 


2 


3 


4 


5 


6 


17 . 


Tests and other evaluations reflect the course content/assignments. 


© 


© 


© 


© 


© 


© 


18 . 


The instructor has made clear the basis for grading. 


© 


© 


© 


© 


© 


© 


19 . 


Test items are clearly written. 


© 


© 


© 


© 


© 


© 


20 . 


Tests/other evaluations provide an opportunity to demonstrate achievement. 


© 


© 


© 


© 


© 


© 


21 . 


The instructor provides feedback on tests and assignments. 


© 


© 


© 


© 


© 


© 



For questions 22-24, a “6” on the right side indicates your opinion is “Strong” and a “1” on the left side 
indicates your opinion is “Weak”. 



In my Opinion.... 


i 


2 


3 


4 


5 


6 


22 . 


The quality of instruction was.... 


© 


© 


© 


© 


© 


© 


23 . 


The contribution of the course to my learning or professional preparation was.... 


© 


© 


© 


© 


© 


© 


24 . 


The likelihood 1 would recommend this instructor to others is.... 


© 


© 


© 


© 


© 


© 



A. My student classification is 



Freshman 


© 


Sophomore 


© 


Junior 


© 


Senior 


© 


Graduate 


© 


Other 


© 



B. My cumulative GPA is 

3.500 - 4.000 O 

3.000 - 3.499 O 

2.500 - 2.999 O 

2.000 - 2.499 O 

Less than 2.000 O 
Do Not Have One O 

C. The course is 

Required in my Major O 
An elective in my Major O 
Other O 
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